Conversation
mrge from upstream
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new 'preliminary-testing' workflow. This workflow leverages an AI agent to analyze Jira issues, GreenWave gating status, and GitLab merge request comments to determine if a build has passed preliminary testing. The changes include adding a new make preliminary-testing target, updating the README with detailed documentation for this workflow, and implementing new Python modules for the handler, the AI analyst, and tools to fetch data from Jira, GreenWave, and GitLab. A review comment suggests improving the ValueError message in preliminary_testing_handler.py for better debuggability by including the specific state value.
| class InputSchema(BaseModel): | ||
| issue: FullIssue = Field(description="Details of JIRA issue to analyze") | ||
| build_nvr: str | None = Field(description="NVR of the build to check, if available") | ||
| jira_pull_requests: str = Field( | ||
| description="Pull/merge requests linked in Jira Development section (JSON)" | ||
| ) | ||
| current_time: datetime = Field(description="Current timestamp") | ||
|
|
||
|
|
||
| class PreliminaryTestingResult(BaseModel): | ||
| state: TestingState = Field(description="State of preliminary testing") | ||
| comment: str | None = Field( | ||
| description="Comment to add to the JIRA issue explaining the result" | ||
| ) |
There was a problem hiding this comment.
It would make sense to put these in a separate file.
|
|
||
|
|
||
| TEMPLATE = """\ | ||
| You are the preliminary testing analyst agent for Project Jötnar. Your task is to |
| and not self.ignore_needs_attention | ||
| ): | ||
| return self.resolve_remove_work_item( | ||
| "Issue has the jotnar_needs_attention label" |
| if len(issue.components) != 1: | ||
| return self.resolve_flag_attention( | ||
| "This issue has multiple components. " | ||
| "Jotnar only handles issues with single component currently." |
| | Condition | Action | | ||
| |-----------|--------| | ||
| | All tests passed and Test Coverage is set | Sets `Preliminary Testing = Pass` with a summary comment | | ||
| | All tests passed but Test Coverage is not set | Flags for human attention (`jotnar_needs_attention` label) with test results in comment | |
| tools = [ | ||
| FetchGreenWaveTool(), | ||
| FetchGitlabMrNotesTool(), | ||
| ReadIssueTool(), | ||
| ] |
There was a problem hiding this comment.
I think these could use some constraints. For start, the ReadIssueTool should be called first, without information from the issue, the agent can't progress. At the same time, both FetchGitlabMrNotesTool and FetchGreenWaveTool should only be called after ReadIssueTool, using the only_after parameter.
The rest depends on how can the system behave in practice.
For example, if an issue has at most a single build attached, the FetchGreenWaveTool should be limited to a single call.
Similarly, unless there is a reason why a single issue wouldn't be enough, the ReadIssueTool should be limited to exactly one call. Because after the information about issue is retrieved, subsequent calls are, at best, very unlikely to provide any more.
Implement a new `preliminary-testing` supervisor workflow that automates the evaluation of preliminary testing (gating and CI checks) for RHEL Jira issues. Previously, QE engineers had to manually verify test results and set the Preliminary Testing field — this agent handles it automatically. The workflow uses an AI agent (BeeAI ToolCallingAgent) to analyze test results from two sources: 1. GreenWave gating status — fetches and interprets the HTML page from gating-status.osci.redhat.com for the build NVR when available. 2. OSCI results in MR comments — discovers linked merge requests via the Jira dev-status API, then fetches MR notes from GitLab to find OSCI "Results for pipeline ..." comments. The workflow gracefully degrades when only one source is available (e.g. no build NVR set yet, or no linked MRs). Entry conditions: issue must be In Progress, Preliminary Testing not already Pass, and at least one of Fixed in Build NVR or linked PRs. Outcomes: - Tests passed + Test Coverage set → sets Preliminary Testing = Pass - Tests passed + Test Coverage missing → flags for human attention - Tests failed/not running/error → flags for human attention - Tests running/pending → reschedules for later New files: - supervisor/preliminary_testing_handler.py — main workflow handler - supervisor/preliminary_testing_analyst.py — AI agent for analysis - supervisor/tools/fetch_greenwave.py — BeeAI tool for GreenWave HTML - supervisor/tools/fetch_gitlab_mr_notes.py — BeeAI tool for MR notes Modified files: - supervisor/jira_utils.py — add set_preliminary_testing() and get_issue_pull_requests() (Jira dev-status API) - supervisor/main.py — add preliminary-testing CLI command - Makefile — add preliminary-testing make target - README-supervisor.md — document the new workflow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
martinky82
force-pushed
the
preliminary-testing-agent
branch
from
a491e71 to
c50007f
Compare
2 participants
TODO:
packit/packit.dev.Fixes
Related to
Merge before/after
RELEASE NOTES BEGIN
Packit now supports automatic ordering of ☕ after all checks pass.
RELEASE NOTES END